• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    Procedure and guidance system for automatic horticultural product collection based on 3d digital modeling. The invention consists of a procedure for the selection of fruit and vegetable products comprising the creation of a 3d model as well as a system for guiding a robotic manipulator for the collection of horticultural products carried out by said selection procedure. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2540676A1
    申请号:ES201530231
    申请日:2015-02-25
    公开日:2015-07-10
    发明作者:Juan Francisco REINOSO GORDO；Francisco Javier ARIZA LÓPEZ
    申请人:Universidad de Granada；Universidad de Jaen；
    IPC主号:
    专利说明:

    Procedure and guidance system for automatic collection of horticultural products based on 3D digital modeling APPLICATION SECTOR OF THE INVENTION
    The present invention is part of the technical sector of agricultural mechanization and involves the practical application of three-dimensional digital modeling techniques. STATE OF THE TECHNIQUE
    Automation in the collection of agricultural products
    The automation of the collection of fruit and vegetable products is one of the aspirations of agricultural mechanization. However, even today there are crops and products, such as green and purple asparagus, whose mechanization has not been successful. In these crops, manual harvesting is still the majority due to the costs of the harvesting machines and the loss problems they present.
    The growing market demand and the rise in labor costs over the years has led some crops to be abandoned in countries such as Spain and relocated in geographical areas with lower labor costs.
    Regardless of the cost of labor, the manual collection of fruit and vegetable products often requires awkward postures that cause ailments, risks and extreme fatigue in the collection operators.
    The two previous facts are especially critical in those cases in which the harvest cannot be carried out in a single pass. In this case, more labor is required and that the collection of each pass does not damage the product that is pending development for the next collection pass.
    Since the middle of the last century there are numerous initiatives to automate this collection. There are proposals ranging from semi-automation to full automation. Thus, in ES2014148 a harvesting machine is described that opens and closes the ridges where the white studs (shoots or turiones) are located, so that in the middle of the opening and closing process they are cut and picked manually. Other patents such as US4,288,970 and US4,918,909 describe fully mechanical selection and cutting systems, in which the selection is always made by the height of the stud and the cut by entering the stems in narrow streets where electromechanical or electro-optical sensors are Those responsible for giving the cut signal.
    ES2148654 T3 (EP96116737.6) describes a system focused on determining the position for the case of white asparagus not yet emerged and based on sensors with underground detection capacity since white asparagus is cultivated below ground level, buried, so that sunlight does not affect him. In this case, infrared techniques, ultrasound and a marker or physical position signaling system are proposed.
    In US8,136,336 a selective asparagus collector is described as having a sensor-collector system that operates in relatively narrow streets or bands. Each sensor commands a mechanical collector, so the system is complex and expensive.
    There are also collection machines on the market, an example is the Geiger SP-2012 combine harvester (Geiger Manufacturing, Stockton, California, USA), but its losses (from 20% to 40%) are still high compared to manual collection, they generate collateral damage to the remaining turions and their price is high. In this case the detection of the turions is carried out by means of two laser beams that determine if the height of the stud is adequate and an optical head with 16 channels, each of three inches, which correspond to 16 cutting cylinders.
    In EP0053994 B1 a fully automated robot is described in the movement and cutting, and whose sensor part consists of a camera and a light source, both opposite and parallel to the horse. In this case the position of the stud is obtained from two shots of the camera in different positions that allow to determine the movement of the robot until it is perpendicular to the turion to make the cut. It also describes the possibility of using two cameras, one next to the other, with different amplitude of the field of vision and persistence to achieve a double shot to determine the position of the stud and command the movement and cutting of the robot.
    Irie et al [Irie N., Taguchi N., Horie T., Ishimatsu, T. (2009), Development of Asparagus Harvester Coordinated with 3-D Vision Sensor, in Journal of Robotics and Mechatronics, Vol. 21, No.5 , pp. 583-589] describe a carriage on rails that, together with the processing system, has two laser projectors that mark the minimum and maximum height on the stem and a television camera with which they determine if the turion should be cut and also generate the control orders of the robotic arm. Lewis [Lewis, A., (2013), Automated Asparagus Harvester Feasibility Study, in Master of Engineering Management ENMG 680] proposes a shape recognition system (height and width) based on two digital cameras and an artificial lighting system based on LED diodes
    Currently, numerous applications are being developed within the field called precision agriculture. Precision agriculture combines the precise knowledge of the position of agricultural machines through global satellite navigation technologies (GNSS) such as GPS (Global Positioning System), with particular actions
    (eg fertilizer, planting, phytosanitary treatments, etc.) on each position of the agricultural plot or on specific plants (eg on a specific olive tree), for which conventional actuator systems (eg electrical, hydraulic systems, etc.) are used. .
    In WO2007088225, an artificial vision system is described for collecting small fruits in hydroponic crops in high and in rows, in structured environments such as a greenhouse. This invention consists of a vision system that allows the precise guidance of a robotic device for the apprehension, cutting and storage of the fruit, designating the cut point of the peduncle. The basic vision system is composed of: two color cameras, a matrix of specific optical laser diodes and an additional laser diode capable of projecting a flat or fanned optical beam in a way that allows it to differentiate and locate the individual fruits in the clusters or groups from the three-dimensional reconstruction of the spots or marks projected in the cluster by the set of specific optical lasers arranged in matrix. This system has several limitations with respect to the proposed invention: a) it only uses sensors that work in the visible spectrum, which limits the selection of the fruit to a comparison of its geometric and color characteristics without being able to evaluate the real physiological state, b) the system is designed for use in the structured environment of a hydroponic crop without substrate where the fruits are high and spatially pre-organized and where there is an artificial background that facilitates detection this conception is not valid for application in the open field, c) the system organizes the hanging of pendulous hanging fruits (cut point on the fruit) while for the case where this patent is centered the cut point is lower and can be affected by the variability of the land, d ) the system does not generate a 3D model but patches of measurements on fruits, such that this conception is not valid for application in c open time,. e) the camera system is reduced to 2 and its positioning with respect to the fruits would be exclusively “lateral” while in our proposal the number of cameras would be greater and its “enveloping” disposition, that is, we would cover the 360 degrees of surrounding horizon to asparagus before cutting. 3d modeling
    The 3D digital surface models (hereinafter “m3D”) are a numerical and digital representation, of a metric nature, of a reality such as the surface of a land or a building. There are currently 3 types of systems for the creation of m3D: a) passive (cameras), b) active (laser scanners), c) passive / active (depth camera like Microsoft® Kinect®).
    Passive systems
    Within the first type of systems, there are those based on photogrammetry. Photogrammetry is a technique that allows you to reconstruct an object in 3D from a series of photographs taken from different points of view and in which said photographed object appears. This technique has been used since the 1950s (1950) in the field of cartography. However, to apply this technique effectively in the 3D modeling necessary for this application requires knowledge of terrain coordinates to scale and level the model (with the vertical of the model following the direction of gravity). In the last 10 years this technique has been improved with contributions from the field of computerized vision, specifically with the creation of algorithms for the automated identification of relevant point objects, the best known being the SIFT [Lowe, D.G. (2004). Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91-110.]. SIFT means scale-invariant feature transform and thanks to that invariant characteristic of the identified object, the same point can be identified in the different photographs that appear, denominating such points as homologous points. This identification of homologous points will allow to reconstruct the relative position (photogrammetric terminology) existing between two photographs at the time of taking. The terminology used in the field of computerized vision is "structure due to movement" (in English, "structure from motion," or SfM). A quick and usual way to know the relative position between two cameras is by means of the 8-point algorithm (Hartley and Zisserman, 2003), said algorithm is based on the epipolar geometry generated by a plane that contains a point in the model space and to the 2 points of view of each one of the photographs. Once the positions of the two cameras in space are known, that is, their SfM, it is possible to know the position of any point in the model space by knowing their photoco-ordinates, the latter process is known as the resection. Notwithstanding the above, in order to form the m3D (in the absence of scaling and leveling it with respect to a certain reference system), more than 2 photographs are usually necessary. The increase in the number of photographs implies the existence of residues in homologous points that appear in more than 2 photographs, which leads to solving an optimization problem, so that optimization is sought by minimizing waste according to some criteria. The most commonly used criterion is the minimization of the L1 standard (Triggs, B., Mclauchlan, P., Hartley, R., Fitzgibbon, A. (1999). Bundle adjustment — a modern synthesis. In Proceedings of the International Workshop on Vision Algorithms: Theory and Practice. 298–372) and the procedure to achieve this is called beam adjustment (Brown, DC (1976). The bundle adjustment - progress and prospects. International Archives of Photogrammetry 21 (3): 3-03, 33 pages). To achieve a good adjustment, the camera calibration is necessary, and more when this is not a photogrammetric camera, such as the current digital compact, bridge, reflex or mirrorless. This calibration can be done a priori, and incorporate the calibration parameters before the relative orientation or by self-calibration, which involves introducing the restrictions in the beam adjustment system itself (Brown, DC (1976). The bundle adjustment - progress and prospects International Archives of Photogrammetry 21 (3): 3-03, 33 pages, Snavely, N., Seitz, M., Szeliski, R. (2008). Modeling the world from internet photo collections. International Journal of Computer Vision, 80 (2): 189-210). The solution of scaling the model can be done by means of a Helmert transformation that also involves minimum quadratic adjustment, this implies the knowledge of points on the ground or the knowledge of the position of the cameras in their own reference system. In a mechanized system for collecting horticultural products such as the one proposed, this reference system can be provided by the chassis of the harvesting machine.
    Active systems
    Within the active systems there are various technologies: laser, sonar, infrared, etc. These devices are called assets because they are capable of emitting a signal pulse. The measurement of the distance between the emitter and the object is made after receiving the bounce on the wave pulse emitted on the object of interest. Depending on the technology, distance measurement can be calculated in two ways (Pfeifer, N., Briese, C. (2007). Laser scanning - principles and applications. In GeoSiberia): depending on the flight time or taking into account the difference phase In geomatic applications the active devices are usually based on a point light (pointer) where there is a single emitting focus, but in other fields curtain-shaped arrangements are used, which allow sweeping the elements that cross it in great detail. These techniques provide the highest accuracy and offer the highest point density in the shortest time of data capture.
    Passive / active systems
    The principle of Passive / Active sensors is the same as that used in photogrammetry and computerized vision, that is, reconstruction of the structure or position of two cameras from their respective images (SfM), and once the parameters of orientation reconstruct the m3D of the homologous points that appear in these images. The difference is that one of the images is a pattern of points whose coordinates, known by the system, are generated randomly by a program. Said pattern is introduced into an infrared projector whose lens has known calibration parameters. Therefore the infrared projector would act as camera 1 in the SfM. The other difference is that the other image is not captured by a camera 2 in the visible spectrum, if not in the infrared wavelength, which will capture the dot pattern emitted by the infrared projector (MacCormick, (2013) How does the Kinect work http://users.dickinson.edu/~jmac/selected-talks/kinect.pdf (Last accessed February 24, 2015). Due to these two differences with photogrammetry (passive sensors), this is why this category has been referred to as Passive / Active. Passive by the infrared camera and active by the infrared projector.
    Attributes for smart and quality selection
    The systems referred to in the automation section in the collection of agricultural products base the collection exclusively on the height criteria and, in some cases, on the conjunction of the height and diameter criteria. At present it is possible to make a much more selective collection with the purpose of collecting only the product that meets predetermined quality standards. This smart selection is based on various parameters.
    Vegetation indices have been used profusely to characterize the health of many plant products, both for food and plant conservation purposes. Probably the most used index is the NDVI (Normalized Difference Vegetation Index). The usual way to calculate this index is from images taken in the near infrared (NIR) and visible (VIS) spectrum, so that for each pixel a value is obtained
    based on the following formula:. The range of the NDVI ranges between -1 and +1,
    indicating maximum health the closer to +1. The NDVI being an indicator of plant health (MYNENI, RB, HALL, FG, SELLERS, PJ and MARSHAK, AL, 1995, The interpretation of spectral vegetation indexes. IEEE Transactions on Geoscience and Remote Sensing, 33, pp. 481– 486), does not always mean that such vegetable is the most appreciated by the consumer and therefore the highest quality. Other aspects such as organoleptic properties, texture and color will be involved in determining quality. Thus, the set of characteristics that for a market (eg Europe) define a product of the highest quality does not coincide with the set of characteristics demanded by a different market (eg Asia). In the case of asparagus, texture can be expressed as a function of two other variables that are maximum shear force and shear energy, which can be estimated from infrared (Flores-Rojas, K, Sánchez, MT, Pérez-Marín , D, Guerrero, JE, Garrido-Varo, A. 2009, Quantitative assessment of intact green asparagus quality by near infrared spectroscopy. Postharvest Biology and Technology. 52 (3): 300-306). Therefore, the use of an infrared camera can be useful, in a system like the one proposed, when estimating the parameters of plant health and texture. As for the color parameter, a color pattern will be sufficient to characterize and categorize the colors preferred by consumers.
    Thus, to avoid the inconvenience of the collection of existing systems (losses of harvested material, collateral losses in unharvested material, expensive mechanical technologies, unintelligent digital technologies, etc.) and to improve the harvest system taking into account the requirements From the current market and to the possibilities offered by precision agriculture and new distance measurement technologies, a selective collection system based on shape parameters is proposed, by means of three-dimensional digital models, and organoleptic parameters. OBJECT OF THE INVENTION
    The first object of the invention consists of a process for the selection of fruit and vegetable products (hereinafter, "process of the invention") comprising the creation of a 3D model that contains information on the geometric characteristics of the terrain, together with geometric parameters, biological and biological activity of product elements that determine their physiological state.
    Preferred examples of each of these parameters are:
    • geometric parameters: location, height, diameter,
    • biological parameters: weight, color and texture,
    • Biological activity parameters: maximum shear force and cutting energy.
    Some of these parameters can be measured with the appropriate sensors and others are modeled on the basis of the measured parameters and known product information to be collected.
    A second object of the invention is a system for guiding robotic manipulators that allows automated collection of fruit and vegetable products, hereinafter "system of the invention", which comprises the means necessary for the generation of three-dimensional digital (3D) models of the product to be collected and from its environment from which precise and usable information will be generated by a decision-making system that commands automated or robotic cutting means.
    This system allows an optimal selection of the material to be cut, because in addition to the height, diameter and attributes such as color or texture, factors taken into account by the systems described above, it uses information relative to the level of biological activity of the product before performing the cut, so that you can select the material that meets the quality criteria at a given time and leave others on the ground to mature.
    On the other hand, the determination of the 3D digital surface model of the context of the cut (product of interest and neighboring products), allows to optimize the displacements of the cutting means, reducing collateral damage to the material that remains on the ground.
    Additionally, once the cut is made, and before depositing it in the storage boxes, the system can perform the automated classification based on the individual attributes obtained for the cut product, which allows the classification in the collection line itself.
    A third object of the invention is a robotic manipulator comprising the system of the invention.
    A fourth object of the invention is a computer program comprising instructions for making a computer carry out the process of the invention.
    Another object of the invention is a computer-readable storage medium comprising program instructions capable of causing a computer to carry out the process of the invention.
    Another object of the invention relates to a transmissible signal comprising program instructions capable of causing a computer to carry out the process of the invention.
    Thus, this invention is especially useful for the collection of green and purple asparagus but also of any other plant, part of the plant and of any species.
    It represents a remarkable technical advance on the systems that are currently used in the collection since it provides the basis for obtaining a geometric and digital model that can be exploited by any robotic cutting and collection system. An isolated turion is not detected, but the geometry of a scene including the ground is extracted, which allows to reduce collateral damage in the cuts, organize the cut and maintain an adequate record of the operation within the precision agriculture systems. This proposal allows the specialization of the participants in the production of the collection subsystems and allows it to be adapted to existing systems. In addition, the invented system is capable of extracting attributes of commercial interest and taking them into account for the automated cutting and selection of the product, which represents a very remarkable advance over the current system. All of the above allows a significant cost savings and improvement in the quality of the product collected. DESCRIPTION OF THE FIGURES
    Figure 1.-Schematic representation of the front elevation an embodiment of the system of the invention. GPS represents the global positioning means, and the operator interface, As the sensor array, Ae the transmitter array, c the calibration signals, b the robotic arm that performs the cut, m the mechanical structure that supports the system and represents the fruit and vegetable product to collect. The lower arrow indicates the direction of movement of the mechanical structure.
    Figure 2.-Schematic representation of the side elevation an embodiment of the system of the invention. GPS represents the global positioning means, and the operator interface, As the sensor array, Ae the transmitter array, c the calibration signals, b the robotic arm that performs the cut, m the mechanical structure that supports the system and represents the fruit and vegetable product to collect. The lower arrow indicates the direction of movement of the mechanical structure.
    Figure 3.-Schematic representation of the plant of an embodiment of the system of the invention. GPS represents the global positioning means, and the operator interface, As the sensor array, Ae the transmitter array, c the calibration signals, b the robotic arm that performs the cut, m the mechanical structure that supports the system and represents the fruit and vegetable product to collect. The lower arrow indicates the direction of movement of the mechanical structure. DETAILED DESCRIPTION OF THE INVENTION
    Definitions:
    Throughout the present description, it will be understood as "scene" the reality formed by a limited area in space and time and which is captured in its entirety by the system of sensors that make up the invention.
    "Physiological state" means the specific point of the vegetative cycle of the individual elements of a product to be collected at a given time. The physiological state of interest is that which is considered suitable for a given collection and that will be determined by a set of parameters (eg height, diameter, volume, weight, color, texture, biological activity, firmness, etc.) that determine the point Optimal collection with a specific objective (eg near sale, sale abroad, industrial processing, etc.).
    The physiological state of a product element is a specific situation at a given time that depends on the history of the product element and is expressed in a series of observable variables (eg height, width, volume, color, vigor, water content, etc.) and other models based on the observable ones (eg weight, shear force, cutting energy, etc.). The physiological state must be adequate according to the purpose of commercialization (e.g. in proximity, abroad, processed in industry, etc.). And the system that is recommended in this patent allows to determine the physiological state in a precise way thanks to the integration of diverse sensors (color, infrared, determination of geometry, etc.), which to date had not been applied together.
    "Geometric characteristics of the land" means the surface shape of the land in which the products to be collected grow. Among its most important features is its microtopography, the presence of foreign objects (e.g. remains, stones, etc.).
    “Product layout” means the relative spatial position of some products with others in a 3D scene, where the most important aspect is the presence of elements of the scene that act as obstacles (eg in the case of asparagus plant stems and turions ) for the extraction of the product of interest (eg mature) without causing damage to future collections (eg product not yet mature). Invention Procedure
    In a first aspect, the invention consists of a process for the selection of fruit and vegetable products which comprises the creation of a 3D model that contains information on the geometry of the soil surface, together with geometric parameters and parameters of biological activity of the product, such such as location, size, layout, color, texture, physiological state of the product, maximum shear force or cutting energy of the fruit and vegetable product.
    In particular, the geometric parameters of the product used are location, height, diameter, weight, color and texture, and the parameters of biological activity of the product are selected from the group consisting of physiological state, maximum shear force and cutting energy.
    In more detail, we can describe the process object of the invention as a procedure comprising the following steps:
    1st. Generation of a 3D model corresponding to a scene in which the geometric characteristics of the terrain, the location and disposition of the product on the ground and the characteristics of the geometric characteristics and biological activity of each product are identified, based on data obtained through Image-forming sensors that work in the visible and infrared spectrum.
    2nd. Extraction of the parameters corresponding to each product element.
    3rd. Selection for the cut based on previously defined decision rules.
    4th. Cut and harvest.
    Each of these stages is described in more detail below:
    Stage 1.-Generation of a 3D model corresponding to a scene in which the position of the product and its location on the ground is identified.
    • The generation of the 3D model is carried out by processing the information captured by means of image-forming sensors, and, optionally, distance meters placed in an "enveloping" arrangement of the scene to be reconstructed. These means allow obtaining the necessary data to establish the parameters of interest of the elements present in the scene and the situation and relative spatial distribution of each other (context). In particular, it allows to obtain the geometric parameters and the location of each product element present in the scene.
    • Optionally, the data obtained by distance measuring sensors can be incorporated to increase accuracy.
    • Other parameters of interest such as color, texture, physiological state, maximum shear force and / or cutting energy of each product element are incorporated into the 3D model, giving rise to more precise models. These parameters are measurable by image forming sensors working in the visible and infrared spectrum.
    This model provides the geometric and topological knowledge of reality, necessary to carry out the following stages.
    Stage 2.-Extraction of the parameters corresponding to each product element
    The parameters provide information on the geometric and organoleptic characteristics of each fruit and vegetable product individually.
    • The three-dimensional model obtained in stage 1 allows to obtain the parameters that will be taken into account in the selection in a simple and individualized way for each product element.
    3rd. Selection for cutting.
    At this stage the decision is made about the collection or not of each of the product elements present in the scene.
    • The selection is made according to predefined decision rules that contemplate the desirable characteristics in each type of product and are compared with the parameters obtained in the previous stages, individualized for each product element (e.g. turion).
    • As an example, the rules may consist of a set of logical restrictions
    (eg "[height ≥A] Y [diameter ≥D] Y [Texture = T1 O Texture = T2]), or implemented as a set of weights of an artificial intelligence technique (eg Neural Networks, Vector Support Machine, Genetic Algorithms, etc.).
    • The set of rules is previously defined by the user, who can establish criteria and assign specific values relative to the parameters (e.g. diameter, height, color, texture, weight, physiological state).
    4th. Cut and harvest.
    As a result of the previous stage, a specific decision on each product element (e.g. turion) will be available, the decision will be to cut it or not.
    • The cutting coordinates will be known precisely from the 3D model of the scene, generated in stage 1.
    Additionally, after the cut, the selected product can be re-classified according to its characteristics and in accordance with additional rules that would meet the commercial classification.
    In a more concrete embodiment, the procedure comprises the following steps:
    Stage 1.
    • Acquisition of images of the scene, both the visible spectrum and the thermal infrared spectrum and, optionally, distance measurement using distance measuring sensors.
    • Generation, by means of photogrammetry techniques, a 3D model of the scene is generated generated from the data obtained by sensors (placed in an enveloping arrangement around said scene) that work in the visible and infrared spectrum and, optionally, including the data obtained by distance measuring sensors to increase accuracy.
    • Extraction of objects from the 3D model. You get as many objects as bodies appear in the scene plus the surface of the land.
    Stage 2.
    • Determination of geometric parameters (height, diameter, volume, weight) associated with each object identified in the 3D model of the scene.
    • Determination determines the color and texture attributes from the images obtained by the sensor means that work in the visible spectrum.
    • Determination of one or more of the parameters of biological activity (of physiological state, optionally, maximum shear force and / or cutting energy) from the images obtained by means of the image forming sensors that work in the infrared spectrum.
    Stage 3
    • Comparison of the parameters determined in the previous steps with the predefined decision rules and selection of the product elements present in the scene suitable for collection.
    Stage 4
    • Individual cutting and collection of each of the product elements selected in the previous step. System of the invention
    A second object of the invention is a system for guiding a robotic manipulator for the collection of horticultural products comprising:
    • Image forming sensor means that work in the visible and infrared spectrum and, optionally, distance measuring sensor means, to measure the geometric and biological activity parameters necessary to carry out the process of the invention.
    • Information storage means for storing the data collected by the sensor means and the predetermined rules on the appropriate parameters for the collection of a product.
    • Information processing means for processing the process of the invention.
    The new system formed by the robotic manipulator controlled by the system of the invention is also the subject of the present invention.
    Each of the means comprising the system is described in more detail below:
    Sensor means for parameter capture
    The system of the invention comprises the sensors necessary to extract from a scene (and consequently, of the product elements present therein) the parameters that allow classifying each product element and the relative spatial distribution of one another (context). The geometric parameters used are location in the scene, height, diameter, weight, color and texture, and the biological activity parameters are selected from the group consisting of physiological state, maximum shear force and cutting energy.
    The generation of the 3D model is achieved using image-forming sensor means that work in the visible and infrared spectrum and, optionally, distance measuring sensor means.
    In a particular embodiment, the basis of the image-forming sensors are CDD or CMOS systems arranged in matrix form (cameras). They are passive sensors that capture the radiation reflected by the elements present in a scene in a given wavelength range.
    These cameras are used to extract parameters of interest (color, texture, physiological state, maximum shear force and cutting energy). Thus, by way of example, sensors with an actuation window in the region of the visible spectrum will be used to obtain the color. To obtain the physiological state and maximum shear force and cutting energy, sensors with an infrared actuation window will be used.
    In a particular embodiment, in the system of the invention, the image forming sensors can also provide the basic data for the calculation of the 3D model and from this the geometric and context parameters, so it would not be necessary to use distance meters .
    However, in another preferred embodiment, the system of the invention comprises, together with the image forming sensors, distance measuring sensors, preferably based on scanning systems to provide the base data for the calculation of the 3D model and from it geometric aspect and context. Examples of sensors based on scanning systems are point beam laser scanners and laser curtain scanners.
    Preferably, the system will comprise a plurality of each of the sensor means, in particular, a plurality of image-forming sensors and / or a plurality of distance measuring sensors, in order to have several views on the scene. The quantity and type of the sensors and their arrangement may vary depending on the product to be collected and the geometry of the collection machine that hosts this system of the invention.
    In another more particular embodiment, and to improve its performance, the system of the invention also comprises structured light generating means. The structured light facilitates the calculation tasks and allows to obtain a greater precision in the creation of the 3D model of the scene. Examples of structured light generating systems are laser or infrared sources that include a diffraction grating which originates the pattern or structure projected by the beam. In the case of fruit and vegetable products that do not have a surface with an irregular texture, such as asparagus, the use of structured light generators is essential.
    If the system uses image-forming sensors, in another preferred embodiment, and to improve its performance, the system of the invention may also include artificial light generating means to illuminate the scene. These systems, based on any type of light source (e.g. Led, halogen, incandescent, fluorescent), will allow for a homogeneous level of illumination throughout the scene. This system allows working in hours of low solar intensity, provides total independence from existing solar lighting, reduces shadows and favors the creation of the 3D model by photogrammetric techniques.
    In another particular embodiment, preferably if the system uses image forming sensors to generate the 3D model, the system of the invention also comprises means for self-calibration. These means are passive devices whose surface has a geometric pattern with high contrast. Media are active systems that perform the same function as brands.
    In another particular embodiment, the system of the invention also comprises passive and active means for maintaining the cleaning of the sensors. Passive media are screens and encapsulations that prevent the entry and deposition of dust and dirt on the sensor and emitting surfaces. Active media are devices of any kind (e.g. mechanical sweeper, pneumatic blower, electrostatic crown, etc.) that clean, sweep or repel dirt.
    Means for storing information
    The system of the invention requires means for storing the decision rules of the method of the invention and, additionally, the information generated by the sensors and for storing system operation records. The means for storing data may be magnetic, optical, magneto-optical or solid-state devices, as in the case. Given the work environment in which the collection activity takes place, solid state media is the preferred option.
    Information Processing Means
    Together with the sensing means, the system of the invention comprises information processing means. The information processing may be carried out by any type of means or device equipped with a central processing unit (e.g. computer, microcomputer, microcontroller, etc.) that is programmable. The purpose of these means is to execute a computer program comprising instructions to make a computer carry out the process of the invention.
    The information processing means will have the appropriate inputs to the type of sensors implemented by the sensor system. This system will be responsible for generating the 3D digital model of the observed scene, both the ground and the flight (objects present in the scene). The processing system will extract the objects present in the scene and their geometry and derived attributes (height, width, volume, context).
    These processing means communicate with the sensing means and with the collection means. Communication can be done via cables or wirelessly by transmitters and receivers of waves (Wi-Fi or Bluetooth, for example), which would allow a remote location of these processing means.
    Once an element that complies with the established decision rules has been identified, the digital processing system will send the appropriate means of movement to the cutting and collection means so that it reaches the cutting position of the element to be cut taking into account the geometries of both element to be cut like the rest of the elements of the scene (context).
    In a particular embodiment, these means may also be used to calibrate and adjust the operation of all the elements involved in the implementation of the method of the invention. In particular, they can be used to calibrate and adjust cutting and harvesting means or the displacement and position control system. The introduction of these parameters can be done through any computer device (e.g. tablet, computer, etc.) through an interface and the appropriate software.
    This digital processing system will have a clock system that allows synchronization of all elements of the system of the invention (sensors, processing) and those provided by the host system (cutting arm, displacement).
    Other considerations about the system
    This intelligent system is based on sensors that are much cheaper and simpler than the totally mechanical or electromechanical technologies. In addition, when using sensors with standard outputs they can be easily replaced. Having a central process system that is responsible for the intelligence of the system allows it to be parameterized, which allows it to adapt better to the needs of each farm (eg species, variety), soil characteristics, characteristics and tastes of the market, etc.
    The information processing and storage means may be separated from the sensors and means for cutting and storing the product. In this case, the system must also have means for transmitting signals that allow communication between the parties.
    Optionally, the system of the invention should allow interaction with the operators of the system. For this, a communication channel (e.g. USB output, RS232) or an interface for human interaction (e.g. LCD, keyboard, screen, touch screen) will be available.
    Optionally, the system of the invention should allow the update of its installed logic system (firmware). To do this, you will have a communication channel (e.g. serial, UART TTL, TWI, SPI, I2C).
    Robotic manipulator comprising the system of the invention
    The system of the invention is designed to guide and control robotic manipulators for the collection of fruit and vegetable products. Examples of manipulators are the cutting arms used in robotic agriculture.
    The assembly formed by a robotic manipulator and the system of the invention gives rise to a new improved robotic manipulator that is also the object of the invention.
    Additionally, the robotic manipulator is complemented with means for the storage of the collected product. These means can be any set of pans, boxes, or rails on which the cutting system can place the cut and classified product.
    A scheme of the arrangement of the different means comprising the system supported on a mechanical structure comprising the mechanical manipulator can be seen in Figures 1, 2 and 3.
    Implementation of the method of the invention
    A fourth object of the invention is a computer program comprising instructions for making a computer carry out the process of the invention.
    The invention encompasses computer programs that may be in the form of source code, object code or intermediate code between source code and object code, such as partially compiled form, or in any other form suitable for use in the implementation of the processes according to the invention. In particular, computer programs also encompass cloud applications that implement the method of the invention.
    These programs may be arranged on or within a support suitable for reading, hereinafter, "carrier medium" or "carrier." The carrier medium can be any entity or device capable of carrying the program. When the program is incorporated into a signal that can be directly transported by a cable or other device or medium, the carrier means may be constituted by said cable or other device or medium. As a variant, the carrier means could be an integrated circuit in which the program is included, the integrated circuit being adapted to execute, or to be used in the execution of, the corresponding processes.
    By way of example, the programs could be incorporated into a storage medium, such as a ROM, a CD ROM or a semiconductor ROM, a USB memory, or a magnetic recording medium, for example, a floppy disk or a hard drive Alternatively, the programs could be supported on a transmissible carrier signal. For example, it could be an electrical or optical signal that could be transported through an electrical or optical cable, by radio or by any other means.
    In this sense, another object of the invention is a storage medium readable by a computer comprising program instructions capable of causing a computer to carry out the process of the invention.
    Finally, a last object of the invention relates to a transmissible signal comprising program instructions capable of causing a computer to carry out the process of the invention. System operation
    Once the decision rules have been defined and introduced into the system of the invention, the system will analyze a particular scene, obtaining the necessary parameters to make the decision on the cutting and collection of the product. If the decision is affirmative, the system will send to the robotic manipulator the appropriate movement orders for the cutting and evacuation of the scene of each selected product element, as well as for the commercial classification of the element already collected.
    Due to the characteristics of the 3D model, the system can send the appropriate movement orders to the mechanical manipulator so that the cut occurs without damage to the rest of the product that remains on the scene. PREFERRED EMBODIMENT OF THE INVENTION
    Next, a preferred embodiment is described, without this implying a limitation on its implementation with other configurations that meet the technical characteristics described above.
    In this exemplary embodiment, the case of using image-forming sensors will be presented.
    The system of the invention is installed in a host system (or "frame").
    Sensor means: The host frame has 8 sensor cameras with a spectral working window in the visible spectrum arranged in 4 arrays of 4 elements (sensor arrays, As) in two horizontal planes. (Figures 1, 2 and 3).
    There are also 4 emitters of structured light beams (emitting arrays, Ae). These foci must be arranged so that the points emitted cover the entire surface of the scene, in a homogeneous manner. A simple geometry like the one shown in Figure 3 makes it possible to have sufficient emission means for the configuration given to the sensor means. The emitters used for structured light used are LEDs due to their low consumption and high light performance.
    Additionally, 4 calibration signals, c, are available within the framework, which can take many different shapes and sizes. In this embodiment the signals are different to facilitate the processing of the images and the calculation of the 3D model. Given the simplicity of this medium, it is preferred that there be the greatest number of them so that each of the cameras observes more than one. The arrangement can be anywhere, but it is preferred that they be associated with other means, such as sensors or emitters, in order to optimize protection and cleaning.
    Storage and processing media:
    The information storage and processing means are properly connected and communicated with each other, with the sensor means and with the means for cutting, collecting and storing the product provided by the host system.
    To facilitate the calculations of the 3D surface digital model generator system, it is important to determine precisely the position and orientation of the sensor devices with respect to the host structure, and with respect to the cutting arm.
    To obtain the absolute position of the robotic manipulator (host system), the use of global satellite navigation systems is preferred, so a GPS (Global Positioning System) system supported by real-time differential corrections has been used. which can be provided with the support of inertial systems. The GNSS clock is the time base used for the synchronization of all processes and devices.
    The preferred processing system is a high-performance CPU given the need to perform complex and agile calculations.
    In the case of the storage media of the information coming from the sensors and the processing, a Flash memory has been used due to its solid state and higher speed.
    Other media.
    To avoid problems due to environmental dirt and dust generated during collection, all chambers, emitting and support means will be isolated from the outside and will have an automated cleaning system. A preferred cleaning system is the use of compressed air jets on the surface of the protective capsule.
    Means for cutting, collecting and storing the product.
    The robotic manipulator guided by the system of the invention consists of a robotic cutting arm and a set of containers to receive the classified product.
    Procedure for generating a 3D model of the scene and intelligent product selection
    The procedure that has been implemented through the system is as follows: Once a scene containing fruit and vegetable products is selected, the following steps are carried out:
    • 8 images are taken of the scene subjected to a structured laser light pattern consisting of dots.
    • Simultaneously at 1, 8 images are taken in the thermal infrared.
    • Through photogrammetry techniques a 3D model of the scene is generated.
    • The 3D model undergoes the extraction of objects. You get as many objects as bodies appear in the scene plus the surface of the land.
    • From the 3D model, each object is determined geometric attributes (height, diameter, volume, weight).
    • The attributes of color and texture are determined from the visible images.
    • The attributes of physiological state, maximum shear force and shear energy are determined from infrared images.
    • From the attributes, the system based on pre-cut decision rules determines the product elements present in the scene that become candidates to collect.
    • The sequence of cuts and cutting paths of each of the product elements is determined in order not to damage the product that will remain on the scene.
    • They are executed one by one, the cuts of each one of the elements of the product. At the same time as the cut is executed, each already cut element is evacuated from the scene and the final selection is made through the post-cutting rules for the commercial classification. The cutting arm will release each item of cut product in a container corresponding to its commercial classification.
    权利要求:
    Claims (13)
    [1]
    1.-Procedure for the selection of fruit and vegetable products that includes the creation of a 3D model that contains information on the soil surface, together with geometric, biological and biological activity parameters of product elements that determine their physiological state
    [2]
    2. Method according to previous claim characterized in that the geometric parameters of the product used comprise the location, height, diameter, the biological parameters comprise the weight, color and texture; and the biological activity parameters comprise the maximum shear force and the cutting energy.
    [3]
    3. Method according to the preceding claim comprising the following steps:
    to.  Generation of a 3D model corresponding to a scene in which the geometric characteristics of the terrain, the location and disposition of the product on the ground and the geometric and biological activity characteristics of each product are identified, based on data obtained through sensors that form image that work in the visible and infrared spectrum.
    b. Extraction of the parameters corresponding to each product element.
    C. Selection for the cut based on previously defined decision rules.
    d.  Cut and harvest.
    [4]
    4. Method according to previous claim in which the first stage the 3D model is generated from the data obtained by image forming sensors that work in the visible and infrared spectrum; and data obtained by distance measuring sensors to increase accuracy.
    [5]
    5. Method according to claims 3 or 4 comprising the following steps:
    • Acquisition of images of the scene, both the visible spectrum and the thermal infrared spectrum and, optionally, distance measurement using distance measuring sensors.
    • Generation, by means of photogrammetry techniques, of a 3D model of the scene generated from the data obtained by sensors working in the visible and infrared spectrum and, optionally, including the data obtained by distance measuring sensors.
    • Extraction of objects from the 3D model.
    • Determination of geometric parameters (height, diameter, volume, weight) associated with each object identified from the 3D model of the scene
    • Determination determines the color and texture attributes from the images obtained by the sensor means that work in the visible spectrum.
    • Determination of one or more of the parameters of biological activity and of the physiological state (optionally, maximum shear force and / or cutting energy) from the images obtained by means of the image forming sensors that work in the infrared spectrum.
    • Comparison of the parameters determined in the previous steps with the predefined decision rules and selection of the product elements present in the scene that are suitable for collection.
    • Individual cutting and collection of each of the product elements selected in the previous step.
    [6]
    6.-System for guiding a robotic manipulator for the collection of horticultural products comprising:
    • Image-forming sensor means that work in the visible and infrared spectrum and, optionally, distance measuring sensor means.
    • Information storage means for storing the data collected by the sensor means and the predetermined rules on the appropriate parameters for the collection of a product.
    • Information processing means for executing a computer program comprising instructions for having a computer carry out the method according to any of the preceding claims.
    [7]
    7. System according to the preceding claim, characterized in that the base of the image-forming sensors are CDD or CMOS systems arranged in matrix form.
    [8]
    8. System according to any of claims 6 or 7, which further comprises structured light generating means.
    [9]
    9. System according to any of claims 6, 7 or 8, further comprising means for self-calibration.
    [10]
    10.-Robotic manipulator for the collection of fruit and vegetable products comprising a system according to any of claims 6 to 9.
    [11]
    11. Computer program comprising instructions for having a computer carry out the method according to any one of claims 1 to 5.
    [12]
    12. Computer-readable storage medium comprising program instructions capable of having a computer carry out the process according to any one of claims 1 to 5.
    [13]
    13.-Transmissible signal comprising program instructions capable of having a computer carry out the procedure according to any one of claims 1 to 5.
    Figure 1
    Figure 2 Figure 3
    类似技术:
    公开号 | 公开日 | 专利标题
    US10761211B2|2020-09-01|Plant treatment based on morphological and physiological measurements
    US10520482B2|2019-12-31|Systems and methods for monitoring agricultural products
    Holopainen et al.2013|Tree mapping using airborne, terrestrial and mobile laser scanning–A case study in a heterogeneous urban forest
    Torres-Sanchez et al.2018|Mapping the 3D structure of almond trees using UAV acquired photogrammetric point clouds and object-based image analysis
    Wang et al.2013|Automated crop yield estimation for apple orchards
    Kattenborn et al.2014|Automatic single tree detection in plantations using UAV-based photogrammetric point clouds
    US7412330B2|2008-08-12|Sensor system, method, and computer program product for plant phenotype measurement in agricultural environments
    Narváez et al.2016|LiDAR and thermal images fusion for ground-based 3D characterisation of fruit trees
    Reiser et al.2018|Iterative individual plant clustering in maize with assembled 2D LiDAR data
    WO2017011595A1|2017-01-19|Modular systems and methods for determining crop yields with high resolution geo-referenced sensors
    KR20200088319A|2020-07-22|A detector that determines the position of at least one object
    da Cunha et al.2019|Estimating vegetation volume of coffee crops using images from unmanned aerial vehicles
    Belton et al.2019|Crop Height Monitoring Using a Consumer-Grade Camera and UAV Technology
    Nielsen et al.2012|Orchard and tree mapping and description using stereo vision and lidar
    Tejada et al.2017|Proof-of-concept robot platform for exploring automated harvesting of sugar snap peas
    ES2540676A1|2015-07-10|Procedure and guidance system for automatic horticultural product collection based on 3d digital modeling |
    US20210264624A1|2021-08-26|Identifying and avoiding obstructions using depth information in a single image
    dos Santos et al.2020|Determining the leaf area index and percentage of area covered by coffee crops using UAV RGB images
    Paturkar et al.2017|Overview of image-based 3D vision systems for agricultural applications
    Rahman et al.2017|3-D image reconstruction techniques for plant and animal morphological analysis-a review
    Reiser et al.2017|Clustering of laser scanner perception points of maize plants
    Fonteijn et al.2021|Automatic phenotyping of tomatoes in production greenhouses using robotics and computer vision: From theory to practice
    Gregorio et al.2021|Sensing Crop Geometry and Structure
    Bao et al.2016|Infield Biomass Sorghum Yield Component Traits Extraction Pipeline Using Stereo Vision
    Párrizas2021|Optical sensors for crop monitoring-from 2d to 3d reconstruction
    同族专利:
    公开号 | 公开日
    ES2540676B2|2016-02-08|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    DE202004009929U1|2004-06-24|2004-09-16|Meisel, Eckhard, Dipl.-Ing.|Automatically guided harvesting machine for asparagus uses one or more digital laser cameras to optically guide tools for picking up asparagus plants|
    ES2253135A1|2006-01-31|2006-05-16|Universidad Politecnica De Madrid|Computer vision system for picking small row-cultivated fruits|
    US20100300059A1|2009-06-01|2010-12-02|Spencer Kim Haws|Selective Harvester|
    DE102012022569B3|2012-11-20|2014-01-23|Soft Control Gmbh Automatisierungstechnik|Method for controlling asparagus lancing device in wagon, involves releasing pulse during reaching of counter condition during driving of wagon, where asparagus lancing device has desired position transverse to dam for lancing asparagus|
    EP3626043A4|2017-05-17|2021-01-13|Inaho, Inc.|Agricultural work machine, agricultural work management system, and program|
    法律状态:
    2016-02-08| FG2A| Definitive protection|Ref document number: 2540676 Country of ref document: ES Kind code of ref document: B2 Effective date: 20160208 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES201530231A|ES2540676B2|2015-02-25|2015-02-25|Procedure and guidance system for automatic collection of horticultural products based on 3D digital modeling|ES201530231A| ES2540676B2|2015-02-25|2015-02-25|Procedure and guidance system for automatic collection of horticultural products based on 3D digital modeling|
    [返回顶部]